Maxwell's Demon and Maxwell-Boltzmann Distribution 
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In a simple model of Maxwell's demon endowed with a Turing-type memory tape, we present an 
operational derivation of the Maxwell-Boltzmann distribution in the equilibrium statistical mechan- 
ics. It is based solely on the combined gas law of the elementary thermodynamics for the model 
of the memory. Equilibrium is defined in terms of the stability of thermodynamic work F against 
noise, where F is the work surplus when resetting the gas system and the memory. This model 
can be applied to non-equilibrium processes, in principle, because of the universality of the Turing 
machine. We demonstrate the dissipation-fluctuation as a simple example. 

PACS numbers: 89.70.Cf, 05.20.-y 



I. INTRODUCTION 

In our previous paper [l[ , we investigated the relation- 
ship between the information theoretic and thermody- 
namic entropies, which was originally discussed by Bril- 
louin in a general context 0, H| and later clarified by 
Landauer and Bennett in the study of the physics of 
computation. In the resolution of the Maxwell's demon 
paradox based on the Szilard engine model Q, the link 
between the two entropies vividly manifested in the form 
of the information erasure principle 043- Generalizing 
Landaucr-Bcnnctt's argument, we showed that the opti- 
mal work cost of information erasure for arbitrary prob- 
abilities is indeed exactly equal to the amount that keeps 
the second law of thermodynamics inviolated. In order to 
do so without relying on the second law, we made use of 
the optimality of Shannon's data compression, which is 
independent of physical laws. Namely, due to the optimal 
data compression, the minimum work cost of information 
erasure was shown to be 



W er = k B T\n2 ■ H(p), 



(1) 



where H{p) is the information theoretic (Shannon) en- 
tropy given by H(p) = -plog 2 p - (1 -p) log 2 (l -p) 
for a binary probability distribution {p, 1 — p}. Also, 
ks is the Boltzmann constant and T is the temperature 
of the heat bath with which the gas is in contact. The 
cost W e i is manifestly expressed in terms of H(p), rather 
than the thermodynamic entropy S(p), hence the clear 
equivalence between the two entropies [l|. 

In this paper, we pursue our information theoretic ap- 
proach to thermodynamics further by enduing the demon 
with a Turing machine. To be specific, we derive the form 
of the Maxwell-Boltzmann distribution, which gives the 



rate of particles possessing energy e in an N particle sys- 
tem in equilibrium, i.e., 



p oc exp 



(2) 
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where T is the temperature of the heat bath that is in 
contact with the particles. 

A significance of the present work is in showing that 
the Maxwell-Boltzmann distribution, which is a key in- 
gredient of statistical mechanics, can in fact be seen as a 
consequence of the optimal reset of memory that records 
the particle states. The memory is modeled solely on 
the basis of the thermodynamic combined gas law. Our 
model of the particle-memory system can also be applied 
to generic non-equilibrium processes and we shall present 
a simple demonstration of the dissipation- fluctuation the- 
orem without u sing statistical mechanics, unlike the orig- 
inal derivation [11] . 

This paper is organized as follows. In Sec. [TTl we reca- 
pitulate the resolution of the Maxwell's demon paradox in 
the case of biased probability distribution [l[. In Sec. lIIIi 
we describe our thermo-Turing machine, in which the 
particles are manipulated by Maxwell's demon accord- 
ing to information in a Turing-machine-like apparatus. 
In Sec. IIV1 we derive the Maxwell-Boltzmann distribu- 
tion in the scenario of the thermo-Turing when the sys- 
tem is in equilibrium. Section [V] is for the generaliza- 
tion of the previous section to systems of arbitrary num- 
ber of energy levels. We shall consider an application of 
the model to non-equilibrium statistical mechanics by re- 
producing the dissipation-fluctuation theorem for a near 
equilibrium case in Sec. ED Section [VTT] is devoted to 
summary and discussions, mentioning an idea that our 
model essentially mimics a Markovian process and that 
universal gates on the Turing-memory side simulate the 
interactions of particles in the system. Appendices are 
for a note on the two entropies and for a comparison 
between our derivation of the Maxwell-Boltzmann distri- 
bution and that by Jaynes [13]. 
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II. ASYMMETRIC SZILARD ENGINE AND 
ERASURE OF COMPRESSED MEMORY 

Let us recapitulate the resolution of the Maxwell's de- 
mon paradox with an asymmetric Szilard engine. The 
engine contains a single molecule gas in the cylinder and 
a partition is to be inserted to divide the whole volume 
V into pV and (1 - p)V with < p < 1 PJ. First, the 
demon inserts the partition into the cylinder and mea- 
sures the position of the molecule. Second, he ties a 
weight at the side in which he found the molecule. The 
demon lets the gas expand isothermally at temperature 
T to push the partition to the end of the cylinder. This 
expansion provides the demon with TS(p) of work, where 
S(p) = ~ fcs{plnp + (1 — p) ln(l — p)} is the thermody- 
namic entropy. Finally, the demon removes the partition 
to close the cycle of the asymmetric Szilard engine. It 
looks as if the demon could extract work from a heat 
bath without leaving any trace in the state of the gas. 
This apparent violation of the second law of thermody- 
namics is called the Maxwell's demon paradox Q. 

The key to the resolution of this paradox is in the fact 
that the demon keeps information on the molecule's po- 
sition in his memory. The physical process of memory 
reset should also be taken into account to complete the 
cycle HH. No costs are assumed for the actions of in- 
serting and removing the partition, tying a weight, mea- 
suring the position of the molecule and computing 
inside the memory Q. It is most natural and judicious 
to consider a symmetric physical configuration for the 
(binary) memory, in terms of transparency of the discus- 
sion. Thus, the necessary amount of work to erase one 
bit of information is k B T\a 2 Q. It is emphasized that 
only the combined gas law is used to evaluate the erasure 
work, avoiding the use of the second law, let alone statis- 
tical mechanics Q . The demon operates the asymmetric 
Szilard engine for sufficiently many times to accumulate 
the data regarding molecule's positions. Then he car- 
ries out the optimal data compression, whose compres- 
sion factor is equal to the Shannon information entropy 
H(p) = — plog 2 p— (1 —p) log 2 (l —p). After compressing 
the data, Maxwell's demon resets the memory according 
to the model we introduced above. 

Therefore, the difference between the work to reset the 
memory and the one extracted by the Szilard engine per 
cycle asymptotically approaches zero, 

k B Tln2-H(p) -TS(p) =0, (3) 

as the length of the data sequence tends to infinity. That 
is, the second law remains intact and the Maxwell's de- 
mon paradox [l[ is resolved. Note that the data cannot 
be compressed in the original case of the symmetric Szi- 
lard engine, i.e., the cylinder with a partition inserted at 
its center Also, Eq. (J3|) strongly corroborates the 

equivalence between the thermodynamic and information 
theoretic entropies in the case of the optimal information 
processing and thermodynamical operation. See more 
details in Appendix. |XJ 



III. THERMO-TURING MODEL 

The primary components of our model are a set P of N 
particles with two energy levels |0) and |1), a long tape 
M, on which a sequence of N memory cells are embedded, 
and Maxwell's demon who are capable of manipulating 
the particles' states and the tape. We will denote the 
energy gap between the two levels of the particles e, and 
assume that each particle is numbered to make a corre- 
spondence with a memory cell. The tape of memory cells 
can be thought of as the one we typically consider in the 
context of Turing machine. Each memory cell can store a 
binary information, either or 1, and it can be modeled 
as a one-molecule gas with a partition at the center of 
cylinder. The whole system P + M is supposed to be in 
contact with environment (heat bath) of the temperature 
T. 

Suppose that initially the particles are all in their 
ground state |0) and the values stored in the memory 
are all 0. Some mechanism, whose detail is irrelevant to 
the present discussion, excites some of the particles to 
|1), and the demon changes the information in the tape, 
monitoring the state changes of the particles. Let p be 
the rate of the excited particles, i.e., pN particles are in 
the excited state, while (1 —p)N in the ground state. 

Let us consider the amount of work F that the entire 
system (P + M) can potentially exert towards the out- 
side, when we let it return to the original state, where all 
particles are in the ground state and the memory tape 
contains only a sequence of 0's. The energy stored in P 
contributes to F positively, and its amount is E := pNe. 
On the other hand, in order to make the data on the tape 
all 0, we need to consume some energy W eT , as a cost to 
erase information. As a result, we have 

F = E - W eI . (4) 

Since we are naturally interested in the optimal (largest) 
value of F for a given p, W cr needs to be minimized. 
Thus, we have W el = NH(p)k B T\n2 [lj, which leads to 

F = pNe — NH(p)k B T In 2. (5) 

We shall define the equilibrium to be the state whose 
F is stable against small noises on either the tape or 
the particles. Let us now describe a physical scenario 
that justifies the consideration of the stability of F as 
condition for equilibrium. Because we now focus on the 
stability of the equilibrated state, we imagine a process 
in which the state starts from equilibrium and a small 
number of errors occur. The condition of the equilibrium 
requires that the particle system stays in equilibrium un- 
der the repetition of this process. So, in the initial state, 
there are pN particles in the excited state and all the 
memory cells of the tape store 0. 

For the following scenario, we need a Carnot engine C 
that is operated between two heat baths, i.e., those of two 
different temperatures, Th and Tl{< Th)- The engine 
C is used to pump up the energy level of particles by 
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providing work taken from heat baths. The temperatures 
are thus chosen so that the work by C is equal to e. In 
other words, they should satisfy Th/Tl = Qh/Ql and 
e = Qh — Ql, where Qh is the amount of heat flow 
from the heat bath of Th to the engine and Ql is that 
to the heat bath of Xl from the engine. For simplicity, 
we will assume that the expansion (compression) rate of 
the volume is 2 (1/2), thus e = k B (T H - T L )ln2. 

Another component is a work reservoir, which is a stan- 
dard gadget in thermodynamics and typically modeled 
by a spring. It is a mechanism that can keep mechanical 
energy, and we can either extract energy from it or add 
extra energy to it when necessary. The net change in the 
stored energy in the work reservoir should be zero when a 
thermodynamic cycle is completed. (If an excess energy 
was still stored at the end of a cycle, it can be discarded 
as heat into the heat bath.) 

We then consider the following cycle, consisting of 
steps (a)-(g) (see Fig. [TJ). The particles are initially in 
equilibrium with the heat bath of temperature T: pN 
of them are excited, and all of N memory cells contain 
0. Since the particles should always be in a reasonable 
physical state when seen from the outside, all artificial 
manipulations in the cycle are designed to be performed 
on the memory tape; the tape is subject to only occa- 
sional (natural) errors. We also assume that the demon 
knows the value of p, having operated the system for a 
long time. 

(a) The demon lets NH(p) memory cells expand to the 
end isothermally at temperature T. Each cell docs 
ksThi2 of work, and the demon stores the energy 
of W\ = iVif (p)fcgT In 2 in the work reservoir. 

(b) The demon inserts a partition at the centre of the 
cylinder of those NH(p) cells, so that a half of 
NH(p) cells represent the value and the other 
half have 1 . 

(c) The demon applies data decompression, the inverse 
operation of Shannon's data compression, on all N 
cells. After the decompression, pN cells will be in 1 
and the rest in 0. Since this operation is reversible, 
it can be designed so that no energy consumption 
is required. 

(d) The demon re-sorts the data sequence, according to 
the states of particles, so that if the fc-th particle 
has the energy Ei (i £ {0, l})thc fc-th cell contains 
the value i. No energy consumption is needed. 

(e) During some fixed time Ai, a small number (<C N) 
of errors would occur in the tape. An error can be 
modeled as a NOT operation applied on a randomly 
chosen cell. 

(f) Detecting the cells where an error occurred, the de- 
mon changes the states of the corresponding parti- 
cles. He transfers energy from the particles to the 
work reservoir and vice versa, depending on the di- 
rection of the change. 
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FIG. 1: (color online) The virtual process for which we con- 
sider the change of F. The V sign in (b) represents a random- 
ized memory state with no physical distinction between and 
1; the molecule can move around in the whole configuration 
space of the cylinder. The demon operates both the memory 
tape and particles, sitting (standing) in between them. 



(g) The demon erases all data in the tape, which now 
stores H{p') bits with p' being the new rate of l's. 
The minimum work is Wi = Ai7(p')fcsT In 2, a 
part (or all) of which would be supplied by the work 
reservoir. 

The data sequence returns to 00 ... 0, while p'N par- 
ticles are now in the excited state. The random NOT 
in the Step (e) may be a physical error to particle state, 
rather than a computational error to the memory state. 
Such a physical error can be triggered by thermal fluc- 
tuation, thus the energy for flipping the state is supplied 
by its environment in this case. In the above scenario 
(Steps (a)-(g)), on the other hand, a computational error 
occurred to the memory without energy supply, and then 
the demon changed the particle state accordingly by tak- 
ing energy from the heat bath. So the net physical effect 
and the energy balance are the same. 

The detection of errors in the Step (f ) may seem to re- 
quire an additional memory, which is as long as the tape, 
to keep the data in the previous cycle. If so, we would 
need to take another erasure cost as well, making the 
story cumbersome. Fortunately, the demon can detect 
errors simply by comparing the data stored in a cell and 
the state of the corresponding particle. The necessary 
number of memory registers is negligible compared with 
the length of the tape. 
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One might naively wonder why we do not simply en- 
code the particle state to the tape without the steps (a)- 
(c) . The point is the energy W\ that we can extract from 
the heat bath exploiting the initial low entropy state. 
Although it may look unnecessarily complex, we believe 
that it is the most natural scenario in which the stability 
of F manifests as a condition for equilibrium. 



IV. MAXWELL-BOLTZMANN DISTRIBUTION 

There is a one to one correspondence between the phys- 
ical state II in the phase space of classical particles of P 
and the computational state ^ of the tape of the Turing 
machine M. This correspondence is guaranteed in the 
two ways. That is, II — )• \f by measurement of II and its 
encoding in 'J, and ^ — > II by readout of ^ of the Tur- 
ing machine and a subsequent operation of the Carnot 
engine C. 

Equilibrium is characterized by the balance between 
the energy income and expenditure in P+M against small 
errors to \&, which we model by random operations. Ar- 
bitrary operations can be simulated by NOT and Toffoli 
gates, since any reversible computation can be executed 
by these two. Once a NOT or Toffoli operation occurs to 
a randomly chosen cell, the demon activates the Carnot 
engine C to change the state of the corresponding parti- 
cle. 

Let us start with an intuitive picture of equilibrium. 
Suppose there are more O's than l's. Then random NOTs 
tend to equalize the populations of O's and l's in the tape. 
As the number of O's approaches that of l's, the demon 
has to consume more energy to excite particles from |0) to 
|1). At the same time, the bit string on the tape becomes 
less compressible, thus larger information content, which 
means that more work would be needed to erase it. This 
energy for erasure can be paid by the particle system. 
So the whole system can be said to be in equilibrium, 
if these increases in energies for state excitation and for 
information erasure are equal. 

The net energy balance in the process from (a) to (g) 
is simply a sum of the energy transfers in the Steps (a) , 
(f), and (g). It is 

Wi + (p'N - pN)e - W 2 = Ap ■ Ne - NAH(p) ■ k B T In 2, 

(6) 

where Ap = p' - p and AH(p) = H(p') - H(p). The 
above cost is equal to the change AF of the function 
introduced in Eq. ([5]). Since the number of errors is 
small, i.e., Ap <c 1, AH(p) = dH(p)/dpAp = H'{p)Ap. 
When the system is in equilibrium, AF = 0, thus 

e - k B T ln2 ■ H'(p) = 0. (7) 

The same equation holds in the case where the NOT gate 
hits a cell of 1 to change it to 0. 

Noting that In 2 • H'(p) = ln[(l -p)/p), we see that Eq. 



(JT]) reduces to 

which is nothing but the Maxwell-Boltzmann distribu- 
tion. A similar calculation can be done for the case in 
which the Toffoli gate is considered as an error operation. 

To sum up the Maxwell-Boltzmann distribution (|8|) has 
been operationally derived under the assumptions 

(1) the state preparation by the Turing machine and 
the device (Carnot engine) C to assign energy levels 
for the A-particlcs 

(2) Landaucr-Bcnnctt's principle for the erasure of 
memory 

(3) the equilibrium condition in terms of the work ex- 
tractable from P + M . 

as well as the combined gas law in elementary thermo- 
dynamics. 

Let us now evaluate the second order effect. A straight- 
forward calculation shows that the average change in the 
information-theoretic entropy due to a random NOT is 

H(p)^H(p) + H\p) { -A-^+H"(p)^ + --- (9) 

noting that the transition — > 1 takes place with prob- 
ability 1 — p, which increases j) to p + 4, while the 
transition 1 — > occurs with probability p, which de- 
creases p to p — jj. This makes a change of H(p) — > 
(1 — p)H(p + jj) + pH{p — -If), each term of which can 
be expanded as in Eq. 

The total cost change by the random Toffoli gate is 
computed as 

AF(Toffoli) = -k B Tp 2 H"{p) 1 ^ + O (Jp\ . (10) 

Note that the first order contributions, i.e., the terms 
having H'(p), cancel out due to the equilibrium condi- 
tion. The effect of the random Toffoli gate is essentially 
equivalent to that of the random NOT gate except for 
the overall probability factor p 2 for having two l's in the 
control bits. 

Therefore, the sum of the second order contribution of 
the random NOT and Toffoli gates is 

AF = -fc B T[#(NOT) + #(Toffoh)p 2 ]H"(p)^| 

+<?(^)(>o). (ii) 

That is, the change AF by the random NOT and Toffoli 
gates tends to strengthen the local stability of the cost 
function F near the equilibrium. This further supports 
our definition of equilibrium in the thermo- Turing model. 
It is also noted that the quantity in the square bracket is 
the effective computational complexity. 
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V. GENERALIZATION 

The above argument to derive the Boltzmann distribu- 
tion can be generalized to the case of dits. That is, the 
cells of the tape can store d values from to d — 1 , and 
the particles have d energy levels from \E ) to \E4_1). 
Let Pi be the ratio of the number of cells storing the 
value i to the total number of cells on the tape. Sup- 
pose that the fc-th particle is excited to the state \Ei) 
when the fc-th cell stores the value i. This state prepara- 
tion can be completed by a similar mechanism to the one 
above, where a Carnot engine C is operated depending 
on the data in cells. Here, we can think of d — 1 engines, 
Q (i S {1, d— 1}). The engine Q is switched on when 
the measurement apparatus D finds i in a cell, and then 
Cj dispenses energy Ei to the corresponding particle. 
Each engine thus utilizes two heat baths of temperatures 
T$ and T { L l) (< ) so that E. t = k B (T^ -T® ) In 2 O , 
where all engines are assumed to be operated between 
two volumes, V and V/2. We will use another notation 
C,^ 1 to indicate the engine Q operated in the reverse 
direction. 

Instead of the random NOT studied in the previous 
section, let us suppose random SWAP operations that 
change the value in a single cell. Let SWAPjj denote 
a SWAP between two values i and j, namely, SWAP,j 
maps a value i of a randomly chosen cell to j and vice 
versa. Note that the NOT operation between two levels 
is effectively the same as the SWAP between them. Once 
such a SWAPjj is applied to the fc-th cell, which stores 
the value i (assume i < j without loss of generality), the 
mechanism D activates C" 1 and then Cj to change the 
state of the k-th particle: C" 1 retrieves energy Ei from 

the particle and discard it to the heat bath at T£ , and 
Cj gives Ej of energy to it. 

Suppose that a SWAP^ has occurred to one of the 
cells. The SWAPjj changes the value stored in the cell if 
it retains either i or j, otherwise nothing happens. The 
probability of such a 'successful' SWAP ij is pi + Pj- If it 
was i the corresponding particle is excited from \Ei) to 
\Ej), and if it was j the particle loses energy to become 
\Ei). Thus once the SWAPy hits a single cell on the 
tape, the energy change in the particle system is {Ej — 
Ei){pi -Pj)/{Pi+Pj) on average. 

The change in the erasure entropy times temperature 
after the single SWAP^ is 



Nk B T In 2 
Pi + Pj 



P, 



Pi + Pj 





1 




-H ( Pi - 

\ 

















1 

N 



k B T\n2— 



Pi + Pj Pj 



(12) 



where H {pi ± jj,pj =p j?) are - ^Z k Pk log 2 Pfc with the 
replacement of Pi and Pj with Pi ± and Pj^fjj, respec- 
tively. 



Making the change in F equal to zero as in the case 
of bits and two-level particles, we arrive at the desired 
relation: 



(Ej - /./,;^_ - keT^-^- In ^ = 
Pi + pj pi + p, p 3 

Pj ( Ej - Ei 

= exp J —— 

Pi \ k B T 



(13) 



This relation holds for any pairs of i and j, hence pi oc 
exp(-Ei/k B T) for all i. 



VI. DISSIPATION-FLUCTUATION THEOREM 

In principle, our thermo- Turing model can be ap- 
plied to generic non-equilibrium processes. Here, we will 
present a modest step to this direction, choosing a par- 
ticular model which exhibits a characteristic feature of 
dissipation-fluctuation theorem. 

The maximal work F we can obtain from P + M to let 
it return to the standard state 00 ... is given by 



F = pNe - Nk B Tln2- H(p) 



(14) 



as in Eq. ([5]). 

Let the change of the maximal work F by an operation 
oj be 



A^F = A u pNe - Nk B Thx2 ■ H\p)A U] p. 



(15) 



Suppose that this change is caused by an external energy 
supply u; 



A U1 F = 



(16) 



The solution to this equation gives a non-equilibrium 
steady state distribution 



Po{u) 



1 



exp (s0 



(17) 



for A^p = 1 /N which corresponds to the transition — > 
1. The distribution becomes po(~u) for A^p = —1/N, 
i.e., the transition of the opposite direction, 1 — > 0. 

Now consider a spatially fluctuating potential u n where 
n (n = 1,2,... N) represents a site location, and we as- 
sume ^2 u n = [141 ] . We then evaluate the average work 
(W) done by a NOT gate under this potential. Sup- 
pose a set of sites m such that po(u m ) > Peq, where 
p eq is the probability distribution for equilibrium. For 
such sites, an incoming energy u n > (n £ {m}) causes 
the change — > 1, and the rate of transition should be 
proportional to the non-equilibrium distribution (the oc- 
cupation probability) 1 — Po(u n ). So the average work 
done is S{«|«„>o}[ 1 ~ Po(u n )}u n - If u n < 0, then the 
induced transition is 1 — > and the average work is 
Sfniu <o} Po(~ u n) u n- Together with a similar consid- 
eration for the set {m'\po(u m ') < Peg}, we have 



(W) 



E 



[1 - Po(Un)]Un + y^^Po(-U n )u 



(18) 
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To simplify the computation, consider the case in 
which the external potential u n is weak. Using the ap- 
proximation po{u) ~ Peg + Pa(0)u, we obtain 



(W) 



k B T 



1 + cxp 



(19) 



where T = ^2 n u n 2 is the sum of squared values of the 
fluctuating potential. Equation (|T9"|) means that the NOT 
gate does negative work, (W) < 0, thus it makes energy 
dissipate to the external space (environment). Further, it 
is proportional to the fluctuation of the potential. It is a 
simple expression of the dissipation-fluctuation theorem 
in the linear approximation of the fluctuating potential 

The readers who arc familiar with the standard 
dissipation- fluctuation theorem would feel more comfort- 
able with the fluctuation in time rather than the spatial 
one of the external potential. In that case, one can re- 
order the site numbers according to the order of occur- 
rence of the NOT gate. Then, the n can be interpreted 
as time, and the average (•) can be understood as that 
over a long time. 



VII. SUMMARY AND DISCUSSIONS 
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Appendix A: Note on thermodynamic and 
information-theoretic entropies 

In our previous paper [l[ , we showed in the scenario of 
Maxwell's demon that the thermodynamic entropy coin- 
cides with the information theoretic entropy in the op- 
timal case of the memory reset. In this Appendix we 
attempt to look at the same problem from a different 
perspective going back to the the basic thermodynamics. 

Let us recall how the thermodynamic entropy was in- 
troduced on the basis of Carnot's theorem. Let the work 
exerted towards the outside be W(i — > f) for an isother- 
mal state change i — > /, which is different from the 
change of the internal energy Ui — Uf, in general. An 
invisible energy flow that contributes to the energy bal- 
ance is called heat Q exchanged between the system and 
the heat bath. Thus, the energy conservation is written 
as 



We have presented an operational derivation of the 
Maxwcll-Boltzmann equilibrium distribution in the sta- 
tistical mechanics by a simple model of Maxwell's demon 
endowed with the Turing machine only on the basis of 
thermodynamics. The crux of the argument is in the 
thermodynamic work needed to reset the memory of the 
Turing-type tape. This model can also be applied to 
non-equilibrium processes. We have demonstrated the 
dissipation-fluctuation using a simple example. 

Our operational derivation can be contrasted with the 
standard one based on the micro-canonical ensemble, in 
which the entropy S is given by the Boltzmann formula 
S = fcslogO(-E) with Cl(E) being the number of states 
under a given energy E. Clearly, we did not rely on the 
micro-canonical statistical mechanics. Instead, the no- 
tion of entropy comes in from information theory, and 
that of temperature is from Landauer-Bennett's erasure 
principle. Jaynes' derivation summarized in Appendix B 
is information theoretic, however, its connection to ther- 
modynamics is rather hypothetical. 

Assuming that the interaction of the particles is math- 
ematically described, e.g., by Hamilton's equations, we 
can say from the universality of the Turing machine that 
the equations of motion are mapped to the operations of 
the machine. This means that the physical interaction 
can be simulated by the gates oj, consisting of NOT and 
Toffoli gates. The relaxation process is simulated by the 
reset of memory (a brute-force Markovian process). Note 
that the memory is also physically modeled on the basis 
of the combined gas law in the elementary thermodynam- 
ics. 



W(i ->f) = Ui- U f + Q. (Al) 

Carnot's theorem claims that the maximal heat flow from 
the heat bath in isothermal process is proportional to the 
temperature T of the heat bath, 

TS th , (A2) 

where the maximization is made over all possible in- 
termediate processes between the initial and the final 
states [lH|. The coefficient Sth is defined as the increase 
of the thermodynamic entropy. 

Now go back to our thermo- Turing model. Clearly, 
our F in Eq. (|4|) is the negative of the work that can be 
exerted outwards, 

F = -W(i->f). (A3) 

Letting the minimum of F in Eq. (| A3|) be equal to Eq. 
©, 

Fmin — ^^max ~U i ~h Uf ~ Qmax 

= peN - NH(p)k B T In 2. (A4) 

Since ~Ui + Uf = peN in our model, we arrive at 
the equivalence between the thermodynamic and the 
information-theoretic entropies; 

S th =NH{p)k B in2. (A5) 

It is interesting to see that both the thermodynamic and 
the information theoretic entropies arc defined as a limit, 
while the former is physically realized in the quasi-static 
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limit and the latter is achieved by the optimal limit of 
the data compression. The quasi-static processes means 
that the operation has to be slow enough compared with 
microscopic processes. This can be viewed as an aspect 
of the Markovian process, in which the information on 
the earlier configuration is lost due to, e.g., the multiple- 
scattering of particles by the cylinder wall. The corre- 
sponding process in our model is the erasure of informa- 
tion, which occurs when the partition in the memory cell 
is removed. To lose the information on the molecule's 
position, we need to wait for a while until it becomes 
impossible to infer the molecule's trajectory. 

As a byproduct, we can see that the extensivity of the 
thermodynamic entropy follows from that of the Shannon 
entropy. Note also that Eq. (|A5|) can be viewed as the 
Boltzmann formula Sth — ks In W, where W = 2 NH ( p ' is 
the number of possible states under a given total energy. 

Appendix B: A comparison with Jaynes' Work 

In the work on the relation between information theory 
and statistical mechanics, Jaynes derived the correspond- 
ing equilibrium state (fT5|) in the following way [f2[. The 
plausible probability distribution is determined by the re- 
quirement that the estimator function Est(pi,p2, • • • ,Pn) 



is maximized under a given average energy (E ) = 
^2iPiEi with ^2jPi = 1. Following Shannon [lOj . he 
required the mathematical properties for the estimator 
Est(pi,p 2 , • • • ,Pn)- (i) Est(pi,p 2 , • • ■ ,Pn) is a continu- 
ous function of Pi, (ii) f(n) '■= Est(l/n, 1/n, . . . , 1/n) 
is a monotonic increasing function of n, and (iii) 
Est(px>P2j • • • jPn) satisfies the composition law. From 
these mathematical requirements (i)-(iii), the estimator 
is uniquely determined to be equal to the Shannon en- 
tropy, i.e., Est(pi,p2, ■ • ■ ,Pn) = - J2iPi log Pi- Maximiz- 
ing the Shannon entropy under the constraints that (E) 
is fixed and XljPi = 1> the plausible probability distri- 
bution is obtained to be the one in Eq. (fl"3)) . Note that 
the temperature does not come in from the physical as- 
sumption but is defined so that the free energy thereby 
obtained coincides with the Hclmholtz free energy. 

In the present work, we have devised a model consist- 
ing of a memory tape (information theoretic object) and 
a set of particles (physical system), both of which are 
in contact with a heat bath of temperature T and are 
operated by the demon. Then, we have defined the equi- 
librium state operationally and subsequently derived the 
Maxwcll-Boltzmann distribution. Thus, the temperature 
dependence of the distribution was naturally obtained, 
whereas it was not so in Jaynes' mathematical work. 
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